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Improving  Platoon  Level  Situation  Awareness  with  Unmanned  Sensor  Teehnology 

Elizabeth  K.  Bowman,  ARL  HRED 
Steven  Kirin,  Mitre  Corporation 


Abstract 

We  investigated  the  eontributions  that  unmanned  sensor  imagery  provides  to  platoon  leaders 
operating  in  a  foree-on-foree  experiment.  We  followed  the  platoon  leader,  platoon  sergeant,  and 
Roboties  Non  Commissioned  Offieer  (NCO)  as  they  eondueted  reeonnaissanee  missions  against 
a  live,  unseripted  opposing  foree  (OPEOR)  in  the  Et.  Dix,  New  Jersey  environment.  Our 
situation  awareness  (SA)  survey  methodology  was  eonsistent  with  the  Situation  Awareness 
Global  Assessment  Teehnique  (SAGAT)  (Endsley,  2000).  We  randomly  asked  leaders  to  answer 
questions  about  the  OPEOR  and  to  draw  their  loeations  on  a  map.  We  later  eompared  these 
answers  and  drawings  with  eomputer-generated  maps  that  showed  geo-refereneed  positions  of 
the  OPEOR.  Our  findings  indieate  that  leader  SA  was  higher,  in  general,  when  using  unmanned 
sensor  teehnology  than  when  relying  upon  human  intelligenee  alone.  However,  the  ability  of 
leaders  to  eonsistently  demonstrate  high  levels  of  SA  aeross  trials  was  disappointing.  We 
believe  this  is  due,  in  part,  to  the  uneven  performanee  of  the  sensor  imagery,  the  periodie  failure 
of  the  eommunieation  system  due  to  dense  foliage  of  the  site,  and  the  inability  of  the  leaders  to 
develop  effeetive  taeties,  teehniques,  and  proeedures  (TIPs)  for  the  persistent  monitoring  of  the 
sensor  images  and  eoordination  with  unmanned  sensor  operators. 


Introduction 

The  C4ISR  On-The-Move  (OEM)  Testbed  was  begun  in  2001  to  allow  demonstration,  testing 
and  evaluation  of  Army  Euture  Combat  Systems  (ECS)  emerging  teehnologies  (RDECOM, 
2005).  The  2005  experiment'  was  the  seeond  instanee  of  a  live  foree-on-foree  test  of  the  systems 
arehiteeture  developed  to  simulate  the  ECS  network^,  and  was  eondueted  at  Et.  Dix,  NJ.  The 
trials  were  eondueted  from  8  August  through  9  September.  Two  foree-on-foree  missions  were 
eondueted  by  Soldiers  aeting  as  test  partieipants  in  this  time  period:  a  base  ease  and  an  advaneed 
ease.  In  the  base  ease  (eondueted  over  3  days)  the  Soldiers,  operating  in  two  platoons,  eondueted 
independent  Intelligenee,  Surveillanee,  and  Reeonnaissanee  (ISR)  missions  in  separate  training 
areas  without  the  use  of  the  ECS  teehnology  (dismounted  battle  eommand  interfaee,  unmanned 
sensors  and  vehieles).  In  the  advaneed  ease  (eondueted  over  5  days),  these  teehnologies  were 
available  to  the  platoons.  The  battle  eommand  interfaee  used  in  this  experiment  was  Eoree  XXI 
Battle  Command  Brigade  and  Below  (EBCB2),  version  4.4.2.  During  the  base  ease,  the  interfaee 
was  only  available  on  vehiele-mounted  sereens.  During  the  advaneed  ease,  dismounted  EBCB2 


*  This  event,  though  eonsidered  an  experiment,  was  more  realistieally  a  struetured  field  observation  where  we  asked 
Soldiers  to  use  new  teehnologies  in  an  operational  setting.  This  allowed  us  to  observe  new  users  and  reeord  how 
systems  performed  in  a  number  of  missions  (Hawley,  2006). 

^  The  Testbed  ineluded  a  set  of  infrastrueture  eomponents  ineluding  a  lab-based  environment  at  Fort  Monmouth  and 
a  field  environment  at  Fort  Dix.  The  former  ineluded  a  modeling  and  simulation  laboratory,  a  network  operations 
eenter  and  a  full  suite  of  the  Army  Battle  Command  System.  The  field  site  ineluded  an  administrative  operations 
eenter,  a  maintenanee  faeility,  presentation  spaee  and  an  extension  of  the  M&S  laboratory.  The  test  site  provided 
position  reporting  deviees,  network  sniffers,  and  audio-visual  eapture  deviees  that  are  integrated  with  the  M&S 
eapabilities.  For  a  detailed  explanation  of  these  eapabilities,  see  RDECOM;  2005. 
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tablets  were  used  by  Soldiers  in  addition  to  the  vehiele-mounted  displays.  This  interfaee  allowed 
the  transmission  of  spot  reports,  free  text  messages,  and  images  ehipped  from  sensor  video.  Also 
used  in  the  advaneed  ease  were  unmanned  aerial  vehieles  (UAVs),  small  UAVs  (SUAVs),  an 
Unmanned  Ground  Vehiele  (UGV)  and  Unmanned  Ground  Sensors  (UGS).  The  SUAV  used 
was  a  hand-launehed  vehiele,  eompared  to  the  larger  maehine  launehed  UAVs. 

Situation  Awareness 

SA  is  the  ability  to  pereeive,  eomprehend,  and  projeet  notions  that  might  ooour  in  the  battlefield 
environment  (Endsley,  2000).  This  is  one  of  the  most  oritioal  and  ohallenging  tasks  in  oombat 
today.  A  vast  portion  of  the  Warfighter’s  job  is  involved  in  developing  SA  and  keeping  it  up  to 
date  in  a  rapidly  ohanging  environment.  All  information  that  supports  SA  must  be  integrated 
into  a  oonoise  pioture  that  supports  mission  aooomplishment. 

The  ECS  C4ISR  arohiteoture  is  expeoted  to  provide  eaoh  eohelon  oommander  with  the  oombat 
leverage  to  ooordinate  resouroes,  engage  and  defeat  the  enemy.  C4ISR  should  provide  a 
oommon  oommunioation  oonneotivity  that  faoilitates  the  land  foroe  oommander’ s  ability  to 
synohronize  elements  to  shape  the  battlespaoe  at  taotioal  levels.  The  assumption  is  that  with 
improved  SA,  the  platoon  oommander  oan  perform  mission  speoifio  tasks  better  and  more 
effioiently. 

The  battlefield  poses  a  variety  of  ohallenges  to  SA  suoh  as  information  overload,  nonintegrated 
data,  rapidly  ohanging  information,  and  a  high  degree  of  unoertainty.  In  our  observations  of  the 
platoon  leaders,  we  were  interested  in  observing  and  dooumenting  the  degree  to  whioh  ECS 
teohnologies  impaoted  situation  awareness. 


In  the  networked  Army  of  the  future.  Soldiers  will  be  required  to  oomplete  several  tasks  at  a 
time,  and  will  have  to  effeotively  shift  among  these  tasks.  Shifting  between  oompeting  tasks 
appears  to  depend  on  four  faotors  related  to  the  tasks  or  the  task  environment:  task  oomplexity  or 
the  oompelling  nature  of  the  task,  eognitive  tunneling  (beeoming  eonsumed  in  a  task),  task 
importanee,  and  physieal  salienee  of  task  properties  or  task  stimuli  (lani  &  Wiekens,  2004).  As 
the  platoon  leaders  interaeted  with  various  teohnologies,  we  observed  and  dooumented  their 
ability  to  attend  to  multiple  tasks  at  onoe. 


Methodology 

Before  the  beginning  of  the  experiment.  Soldiers  attended  a  meeting  to  learn  about  the  goals  of 
the  Testbed  experiment  and  beoome  familiar  with  prooedures.  The  researohers  also  explained 
the  situation  awareness  methodology  and  reviewed  the  rights  of  researoh  subjeots.  All 
partioipants  reviewed  the  volunteer  affidavit  agreement  and  oonsented  to  partioipate  in  the 
experiment.  At  this  meeting,  they  oompleted  a  demographies  questionnaire.  Also  before  the 
experiment,  the  data  eolleetors  responsible  for  the  administration  of  the  SA  surveys  were  briefed 
on  the  survey  format,  administration  prooedures,  and  the  partioipants  they  were  to  assigned  to 
follow. 
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Prior  to  the  force  on  force  trials,  Soldiers  attended  two  weeks  of  training.  This  included 
classroom  training  on  FBCB2  and  field  training  with  the  Unmanned  Ground  Vehicle  (UGV)  and 
the  Small  Unmanned  Aerial  Vehicle  (SUAV). 

During  the  experiment,  SA  was  measured  using  a  modified  Situation  Awareness  Global 
Assessment  Technique  (SAG AT)  (Endsley,  2000).  This  procedure  is  designed  to  measure  an 
individual’s  understanding  at  three  levels  of  perception,  comprehension,  and  projection  while 
focusing  on  critical  mission  objectives.  On  each  day  of  the  experiment,  six  key  leaders 
completed  SA  surveys  at  four  times.  These  were  before  they  departed  from  the  assembly  area, 
twice  during  the  mission,  and  once  immediately  after  mission  halt.  These  leaders  were  the 
platoon  leader  (PL),  platoon  sergeant  (PSG)  and  robotics  NCO  (RNCO)  in  two  platoons.  Each 
survey  contained  questions  that  asked  about  enemy  locations,  size,  and  possible  actions  that 
might  be  forthcoming.  These  were  open  ended  questions,  and  were  designed  by  subject  matter 
experts  (SMEs)  who  were  familiar  with  the  mission  scenarios.  The  surveys  also  contained  a  map 
of  the  relevant  mission  area  and  leaders  were  asked  to  show  locations  of  all  known  Blue  and  Red 
vehicles  and  dismounts. 

Our  data  collection  included  the  manual  recording  of  communication  events  within  one  platoon. 
We  content  analyzed  these  transcripts  to  identify  the  types  of  communication  and  system 
problems  encountered.  The  transcripts  provide  possible  explanations  for  degraded  levels  of 
information  quality  and  situation  awareness  that  might  have  resulted  from  problems  with 
network  connectivity  or  system  breakdowns. 


Results 

Demographics 

In  all,  the  force-on-force  experiment  consisted  of  64  participants.  Thirty  members  from  the  50**^ 
Brigade,  New  Jersey  Army  Reserve  National  Guard  (ARNG)  participated  as  Blue  Eorces  and  34 
contract  employees,  who  were  hired  expressly  for  this  experiment,  participated  as  the  OPEOR. 

A  demographics  survey  was  administered  to  all  participants  prior  to  the  start  of  the  experiment. 
Survey  results  indicate  that  Blue  Eorce  participants  were  highly  experienced  in  that  60%  had 
held  a  leadership  role  and  87%  have  been  deployed.  These  participants  had  an  average  of  9 
years  experience  in  the  Army  and  represented  a  predominantly  enlisted  rank  structure;  all  were 
enlisted  with  the  exception  of  one  ILT.  All  Blue  Eorce  participants  were  male.  Only  41%  of  the 
OPEOR  reported  prior  enlisted  military  experience.  Eor  those  with  prior  experience,  71% 
reported  Army  service.  On  average,  this  experienced  OPEOR  subgroup  reported  8.25  years  of 
prior  enlisted  experience.  Of  the  OPEOR  group  that  reported  prior  military  experience,  24% 
reported  one  or  more  deployments  during  their  Service  careers.  All  of  the  prior  OPEOR  military 
experience  was  in  the  enlisted  ranks.  Two  OPEOR  participants  had  obtained  the  rank  of  E-8,  one 
achieved  E-7,  four  were  E-5,  five  were  E-4,  one  was  E-3  and  one  was  E-2.  Sixty-eight  percent 
of  the  OPEOR  reported  previous  experience  in  the  Eort  Dix  test  range  area.  The  OPEOR  was 
predominantly  male  (88%)  compared  to  12%  female. 
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Transcript  analysis 

The  daily  transcripts  were  content-analyzed  to  determine  how  tasks  were  performed,  by  whom, 
and  what  types  of  problems  arose  in  the  conduct  of  these  tasks.  We  developed  several  categories 
to  cluster  message  content.  These  included  breakdowns  in  equipment,  breakdowns  due  to  user 
issues,  and  FBCB2  tasks.  The  first  equipment  breakdown  category  was  further  divided  to 
include  radio  communications,  FBCB2  vehicle  problems,  FBCB2  tablet  problems,  and 
equipment  problems  with  the  unmanned  vehicles  or  sensors  (e.g.  UAV,  SUGV,  and  UGS).  We 
partitioned  the  category  of  FBCB2  tasks  into  spot  reports,  free  text,  and  viewing  images  from 
unmanned  sensors.  Table  1  shows  these  categories  and  the  number  of  communication  messages 
associated  with  each.  The  number  of  messages  represents  the  communications  events  from  all  8 
days  of  the  experiment. 

Table  1  Factors  Affecting  Task  Completion 


Category 

Messages 

Example 

Equipment  Breakdowns 

•  Radio 

communications 

25 

“Today  was  a  challenge  for  communications  (vegetated  area).  3“' 
squad  could  hear  everyone,  no  one  could  hear  them.  T'  squad  comms 
were  intermittent  at  best.  ” 

•  FBCB2  vehicle 

6 

“FBCB2  screen  keeps  cutting  out,  possibly  due  to  heat  from  engine.  ’’ 

•  FBCB2  tablet 

10 

“1  have  tablet  out,  but  can ’t  send  information.  When  I  put  the  pen  on 
the  screen  it  is  4  to  5  inches  to  the  left  of  where  I  need  to  mark,  the 
cursor  moves  in  its  own  direction.  ” 

•  UAV 

9 

•  Engineers  trying  unsuccessfully  to  open  UAV  spot  report  images, 
believe  it  is  a  technical  failure. 

•  Images  on  spot  report  are  blurred  to  point  of  being 
unrecognizable 

•  SUGV 

1 

When  using  SUGV,  it  was  within  50  meters  of  target  vehicle,  but 
didn ’t  detect  due  to  constraints  of  sensor  movement. 

•  UGS 

1 

PL  is  advised  to  look  at  UGS  field.  Opens  one  image  (takes  13 
seconds).  Opens  second  image  (takes  17  seconds).  PL  misinterprets 
retreating  OPFOR  vehicles  as  reinforcing  force.  Time  pressure  and 
lack  of  regular  monitoring  of  UGS  field  led  to  confusion  with  images. 

User  problems 

•  Dismounted  reports 
to  leaders 

10 

“Any  2''‘‘  squad  element,  this  is  Platoon  Leader,  somebody  get  me  a 
sit  rep.  ”  [from  2"“^  squad]:  “Standby.  ”  “I  need  a  sit  rep  from  anyone. 
Who ’s  next  in  that  chain  of  command?  Squad  1,  is  there  anyone  left 
alive  in  that  unit?  ” 

•  Use  of  subordinate 
team  leaders 

8 

Platoon  leader  stops  RNCO  from  downloading  UAV  images  because 
it  is  interfering  with  his  attempts  to  download  and  view.  Lack  of  task 
management. 

FBCB2  interface 

•  Spot  reports 

15 

All  squads  are  sending  spot  reports  to  PL  from  their  tablets. 

•  Free  text 

4 

PL  sends  priority  information  requirements  report  to  Company 
Commander  in  free  text. 

•  Opening  images 

6 

Images  take  a  long  time  to  open,  often  will  not  open  at  all. 

In  our  review  of  Soldiers’  communications  and  our  observations  during  missions,  we  noted  that 
it  was  extremely  difficult  for  them  to  switch  between  tasks  in  a  coherent  and  coordinated  fashion 
and  that  it  was  nearly  impossible  for  them  to  conduct  several  tasks  at  once.  Task  switching 
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became  a  problem  especially  when  the  PL  did  not  efficiently  use  his  subordinates.  Early  in  the 
experiment,  the  PL  took  responsibility  for  opening  and  viewing  images  received  from  unmanned 
sensors  (this  was  a  responsibility  of  the  Robotics  NCO).  Frequently,  these  images  did  not  open 
right  away  due  to  network  problems.  This  required  the  PL  to  initiate  several  attempts  to  open  the 
message.  Because  the  PL  did  not  monitor  the  unmanned  sensor  images  on  a  continual  basis,  he 
had  difficulty  interpreting  the  image  (e.g.  were  the  vehicles  in  the  sensor  field  reinforcing  or 
retreating?).  In  this  case,  the  RNCO  was  underutilized  and  the  PL  was  susceptible  to  making 
poor  decisions  based  on  uncertain  information.  The  PL’s  efforts  to  download  the  UAV  images 
and  his  interest  in  directing  dismounted  assets  on  the  FBCB2  display  kept  him  from  attending  to 
other  tasks.  We  noted  that  cognitive  tunneling  (becoming  consumed  in  a  task)  appeared  to  be  the 
major  reason  that  leaders  were  unable  to  switch  among  key  tasks  (lani  &  Wickens,  2004). 

Our  observations  showed  that  the  PL  often  became  quite  involved  with  the  FBCB2  screen  in 
attempts  to  manage  the  dismounted  troops.  The  troop  locations  were  marked  on  the  display,  and 
the  PL  used  the  radio  and  free  text  messages  to  coordinate  troop  movement.  Due  to  this 
concentration  on  the  battle  command  interface,  the  PL  neglected  the  unmanned  sensors  and  did 
not  delegate  this  task  to  the  RNCO.  The  RNCO,  following  orders  from  the  PL  to  stay  away  from 
the  sensor  images  (to  avoid  locking  up  the  network),  did  not  stay  in  touch  with  the  sensor 
operators  and  did  not  maintain  an  awareness  of  sensor  images.  Thus,  when  the  PL  was  prompted 
by  experiment  controllers  to  switch  tasks  and  open  up  the  sensor  images,  a  considerable  amount 
of  time  was  needed  for  the  PL  and  the  RNCO  to  understand  the  current  situation.  In  fact,  on 
occasion,  the  PL  misinterpreted  sensor  images  because  no  one  had  maintained  a  general 
awareness  of  sensor  images.  A  more  efficient  use  of  manpower  would  suggest  that  the  RNCO 
might  be  responsible  for  monitoring  sensor  placement,  filtering  images,  and  informing  the  PL 
when  actionable  information  is  received. 

These  examples  suggest  that  the  platoon  members  were  using  unmanned  assets  without  the 
benefit  of  effective  TTPs  designed  specifically  for  this  type  of  scenario.  Procedures  for 
distributing  responsibilities  for  sensor  coordination  among  the  staff  would  have  helped  the 
platoon  leader  manage  the  mission  more  effectively. 

TTPs  for  integrating  unmanned  sensors 

As  we  previously  mentioned.  Soldiers  did  not  receive  field  training  prior  to  the  start  of  the 
experiment  that  would  have  helped  them  develop  TTPs  for  the  use  of  the  unmanned  assets.  As 
the  experiment  progressed,  we  collected  observations  that  will  be  used  to  develop  TTPs  in 
support  of  the  2006  experiment.  Two  key  observations  are  worth  noting.  First,  the  platoons  did 
not  effectively  use  the  RNCO  to  coordinate  the  unmanned  assets  and  monitor  the  images  from 
these  assets.  As  a  result,  sensor  support  was  uncoordinated,  images  were  often  retrieved  late  in 
the  fight,  and  the  images  were  sometimes  misinterpreted.  Second,  we  noted  that  the  unmanned 
air  and  ground  vehicles  were  easily  spotted  by  the  OPFOR,  alerting  them  to  the  presence  of  the 
Blue  Force.  This  knowledge  was  noted  by  the  Soldiers  as  a  reason  to  use  vehicles,  in  some 
circumstances,  as  decoys.  These  TTPs  will  continue  to  evolve  over  future  trials. 


^  On  one  occasion,  when  viewing  an  Unmanned  Ground  Sensor  (UGS)  image,  the  platoon  leader  perceived  that  the 
vehicles  shown  in  the  image  were  reinforcing  the  OPFOR  position  when  they  were,  in  fact,  leaving. 
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Situation  Awareness 


At  the  conclusion  of  the  experiment,  researchers  coded  SA  surveys  according  to  the  two  criteria 
of  fidelity  and  accuracy.  Fidelity  was  defined  as  the  completeness  of  information  provided  in 
answers  to  survey  questions  and  annotations  of  OPFOR  locations  drawn  on  maps.  As  an 
example,  if  a  Soldier  answered  most  questions  with  “uncertain”  or  “don’t  know”  and  the  map 
drawings  had  no  or  few  annotations,  a  low  score  was  assigned.  Accuracy  was  defined  as  the 
degree  to  which  the  leaders  were  able  to  correctly  identify  Red  positions  on  the  map.  We  will 
discuss  the  fidelity  and  accuracy  measures  and  coding  in  turn. 

Fidelity  Analysis 

Initially,  all  surveys  were  evaluated  to  determine  the  fidelity  of  their  responses.  In  this  initial 
review,  the  two  authors  coded  the  surveys  by  examining  the  written  answers  to  questions  and  the 
associated  placement  of  OPFOR  icons  on  the  maps.  A  coding  scheme  was  developed  that 
grouped  responses  into  categories  of  low,  medium,  and  high.  The  grading  criteria  focused  on  the 
respondent’s  knowledge  of  the  OPFOR  as  represented  in  answers  to  the  questions  (strength  of 
force,  planned  actions,  type  of  force),  and  the  detail  provided  of  OPFOR  vehicles  and  locations 
on  the  map  associated  with  the  SA  survey.  A  low  rating  was  given  to  a  survey  in  which  the 
subject  answered  “didn’t  know”  to  75%  of  survey  questions.  A  low  rating  was  also  given  to  map 
annotations  that  were  either  blank  or  had  very  few  identifiable  markings.  The  low  rating 
indicated  that  the  respondent  had  little  understanding  of  the  OPFOR.  A  medium  rating  was 
given  for  those  responses  that  provided  more  granularity  on  the  OPFOR  including  some  level  of 
detail  about  the  enemy  disposition  (locations  of  observation  posts,  target  vehicles,  etc.).  A  high 
rating  was  given  to  a  survey  in  which  the  subject’s  responses  indicated  a  solid  understanding  of 
the  OPFOR  and  map  drawings  differentiated  OPFOR  vehicles,  dismounts,  and  possible  counter 
attack  forces. 

The  analysis  of  fidelity  scores  was  achieved  through  an  independent  and  blind  assessment  by  the 
two  coders.  We  evaluated  the  probes  in  groups  of  10  and  marked  each  as  low,  medium,  or  high. 
Inter-rater  reliability  was  computed  by  calculating  a  percent  of  scores  that  had  been  coded  alike. 
The  first  group  of  20  surveys  resulted  in  an  inter-rater  reliability  score  of  45%.  At  this  point  the 
raters  discussed  their  strategies,  agreed  upon  a  common  scoring  metric,  and  continued  with  the 
additional  surveys.  For  the  remaining  surveys,  an  inter-rater  reliability  rating  of  85%  was 
achieved.  The  first  group  of  20  surveys  was  recoded  according  to  the  raters’  improved 
understanding  resulting  in  a  final  inter-rater  reliability  score  of  95%. 

Of  the  116  surveys  coded  "^,11  (9.5%)  were  graded  as  ‘high’,  28  (24.1%)  as  ‘medium’,  and  77 
(66.4%)  as  ‘low’.  Of  the  1 1  high  ratings,  9  occurred  in  the  advanced  case  runs.  From  the  initial 
set  of  surveys,  we  selected  a  subset  for  further  analysis.  We  selected  those  surveys  that  had  a 
fidelity  rating  that  was  either  medium  or  high,  or  was  low  and  improved  over  the  course  of  the 
mission. 


We  had  a  number  of  surveys  that  were  not  eompleted.  Eaeh  survey  ineluded  four  sets  of  identieal  questions  that 
were  asked  at  random  times  during  the  mission. 
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In  our  analysis  of  fidelity  ratings,  we  used  frequency  distributions  to  compare  SA  fidelity  ratings 
by  case,  role,  and  platoon.  We  were  unable  to  use  inferential  statistics  due  to  the  low  number  of 
subjects  and  the  lack  of  independent  observations. 

Fidelity  ratings  by  Base  or  Advanced  Case 

As  shown  in  Table  2  and  graphically  in  Figure  1,  most  fidelity  ratings  for  the  base  and  advanced 
cases  fell  in  the  low  range.  There  was  a  slight  improvement  in  medium  scores  in  the  advanced 
case,  and  a  larger  improvement  of  high  scores  in  the  advanced  case.  Because  the  fidelity  score  is 
essentially  a  measure  of  how  completely  the  subject  answered  the  survey  questions,  we  can 
suggest  that  the  improvement  in  the  high  scores  during  the  advanced  case  is  possibly  due  to  the 
FBCB2  interface.  This  system  allowed  the  PL  to  track  the  dismounts  and  reported  OPFOR 
locations.  These  percentages  displayed  in  Table  2  are  shown  in  Figure  1. 

Table  2  SA  Ratings  by  Level  of  SA  Fidelity  and  Case 


SAl 

Fidelity  Rating 

Low 

Medium 

High 

Case 

N 

(%) 

N  (%) 

N 

(%) 

Base 

39 

(72%) 

13  (24%) 

2 

(4%) 

Advanced 

38 

(61%) 

15  (24%) 

9 

(15%) 

Total 

77 

(66%) 

28  (24%) 

11 

(10%) 

Fignre  1  Percentage  of  SA  ratings  by  level  of  SA  fidelity  and  case. 


100% 

80% 


Low  Medium  High 

Levels  of  SA  Fidelity 


Fidelity  Ratings  by  Role 

We  examined  SA  fidelity  ratings  for  principal  leadership  roles.  As  shown  graphically  in  Figure 
2,  the  fidelity  ratings  were  fairly  consistent  among  the  three  roles.  In  general,  SA  fidelity  scores 
were  low  for  all  leaders. 
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Figure  2  Percentage  of  SA  ratings  by  level  of  fidelity  and  role. 


The  fidelity  ratings  showed  an  interesting  pattern  by  platoon.  As  shown  graphically  in  Figure  3, 
the  second  platoon  achieved  fewer  low  scores  and  slightly  more  medium  and  high  scores. 

Figure  3  Percentage  of  SA  ratings  by  level  of  fidelity  and  platoon. 


Although  the  Blue  Forces  had  a  fairly  consistent  low  perception  of  the  OPFOR’s  location  and 
strength,  the  fidelity  analysis  indicates  that  Blue  knowledge  of  the  OPFOR  improved  slightly 
during  the  Advanced  Case  trials  when  imagery  from  unmanned  air  and  ground  vehicles  became 
available  and  leaders  were  able  to  use  FBCB2  on  dismounted  tablets. 

Accuracy  ratings 

Accuracy  ratings  for  the  24  surveys  selected  for  the  fidelity  analysis  were  completed  by 
comparing  the  map  annotations  of  the  OPFOR  positions  against  the  ground  truth  grid  coordinates 
that  were  recorded  during  the  experiment  by  controllers.  One  of  the  authors  reviewed  the  map 
drawings  completed  by  the  leaders  and  compared  these  to  screen  shots  from  the  relevant  day  and 
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time  that  the  survey  was  eompleted.  These  sereen  shots  depiet  Red  vehicles  and  dismounts  in 
the  tactical  area,  and  are  the  result  of  GPS  transmissions  sent  from  equipment  installed  in  each 
vehicle  and  each  backpack  carried  by  a  dismounted  Soldier. 

To  grade  the  scores,  the  SME  raters  assigned  scores  in  quartiles  between  0  and  1  (e.g.  0,  .25,  .5, 
.75,  and  1.0).  These  scores  were  based  on  the  number  of  enemy  identified  and  the  percent 
correct.  For  example,  if  a  participant  identified  two  enemy  locations,  and  only  one  was  in  close 
proximity  to  actual  red  placement,  the  resulting  score  would  be  .5. 

The  data  in  table  5  show  the  spread  of  accuracy  scores  when  compared  to  the  fidelity  scores. 
Table  3  SA  Survey  Ratings  by  Fidelity  and  Accuracy  Scores 


SA  Fidelity  Rating 

Accuracy  Scores 

Low  Fidelity 

N  (%) 

Medium  Fidelity 
N  (%) 

High  Fidelity 

N  (%) 

1.00 

1 

(17%) 

3 

(50%) 

2 

(33%) 

0.75 

1 

(20%) 

2 

(40%) 

2 

(40%) 

0.50 

1 

(11%) 

8 

(89%) 

0 

- 

0.25 

0 

- 

5 

(100%) 

0 

- 

0.00 

18 

(69%) 

3 

(12%) 

5 

(19%) 

Total 

21 

(41%) 

21 

(41%) 

9 

(18%) 

These  data  indicate  that  if  a  survey  was  rated  low  in  SA  fidelity  it  would  most  likely  be  judged  as 
inaccurate.  In  general,  participants  exhibiting  low  SA  fidelity  were  unable  to  provide  specific 
information  about  the  enemy  and  were  unable  to  identify  OPFOR  positions  on  a  map.  However, 
several  SA  surveys  were  scored  in  the  medium  and  high  range  for  fidelity,  but  received  an 
accuracy  score  of  “0”.  In  these  cases,  participants  thought  that  they  knew  some  fairly  detailed 
enemy  information,  but  were,  in  fact,  incorrect  in  their  beliefs.  Approximately  20%  of  the 
participants’  surveys  received  a  score  above  .5  on  accuracy,  and  the  majority  of  these  surveys 
received  medium  and  high  fidelity  ratings.  Surprisingly,  five  high  fidelity  surveys  were 
completely  inaccurate  (receiving  a  score  of  0). 

When  we  analyzed  these  data  further,  we  realized  that  in  general  one  platoon  was  over¬ 
represented  in  the  accuracy  ratings.  This  is  a  function  of  generally  low  fidelity  ratings  for  that 
platoon.  The  reader  will  recall  that  the  fidelity  ratings  provided  selection  criteria  for  accuracy 
ratings.  These  low  ratings  could  be  the  result  of  several  factors  at  play  in  the  experimental 
environment  rather  than  internal  to  platoon  participants.  As  the  experiment  progressed,  the 
Testbed  experienced  problems  with  Soldier  participation  (i.e.,  mortality).  ^  Declining  numbers  of 
participants  made  it  difficult  to  fully  field  two  platoons,  and  one  platoon  became  the  fielding 
force  for  the  other.  Though  the  platoon  with  reduced  manning  continued  performing  their 
missions,  they  did  so  at  an  increased  cost  to  their  personnel.  The  fact  that  this  group  was  able  to 


^  The  participants  in  the  experiment  were  members  of  the  NJ  ARNG  and  completed  their  two  week  annual  training 
at  this  time.  However,  due  to  the  length  of  the  experiment  (five  weeks),  the  Soldiers  were  asked  to  extend  their 
commitment  for  extra  pay.  This  commitment  became  difficult  for  some  Soldiers  to  meet  and  resulted  in  declining 
participation  over  time. 
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successfully  complete  missions  speaks  to  the  professional  attitude  and  perseveranee  of  its  leaders 
and  members.  However,  it  ean  be  reasonably  expeeted  that  battlefield  status  information 
reeeived  by  leaders  was  degraded  (e.g.,  redueed  in  number  and  quality,  and  reeeived  less 
frequently)  due  to  a  reduction  in  personnel  and  over-tasking  of  key  leaders.  For  these  reasons 
the  remainder  of  this  seetion  deals  exclusively  with  the  leaders  of  one  platoon. 

As  we  investigated  the  aecuracy  of  SA  in  finer  detail,  we  realized  that  one  leader  in  partieular 
provided  the  best  opportunity  to  test  this  SA  scoring  methodology  and  extrapolate  possible 
understanding  in  relation  to  actions.  We  present  deseriptive  data  on  this  leader  as  an  example  of 
the  best  development  of  situation  awareness  aehieved  in  this  experiment. 

Table  6  shows  the  fidelity  and  aceuraey  seores  by  day  for  the  seleeted  leader.  This  leader  did  not 
eomplete  a  SA  survey  on  every  day  of  the  experiment.  These  data  show  that  the  fidelity  seores 
remain  generally  eonsistent  with  the  exeeption  of  day  5.  The  aeeuraey  seores  for  this  leader 
were  highest  on  days  2,  5,  and  6.  We  suggest  that  the  higher  aecuracy  scores  on  days  5  and  6 
were  due  to  a  combination  of  the  FBCB2  interfaee  available  on  tablets  and  an  improved  use  of 
the  unmanned  sensor  images.  As  the  experiment  progressed,  the  leader  appeared  to  beeome 
more  familiar  with  the  unmanned  teehnology  and  the  images  available  from  the  sensors. 

Table  4  Fidelity  and  accuracy  scores  by  day  for  selected  leader 


Case 

Base  Case 

Advauced  Case 

Day 

1 

2 

3 

4 

5 

6 

Fidelity 

0.40 

0.50 

0.50 

0.50 

0.77 

0.53 

Accuracy 

0.25 

0.50 

0.38 

0.38 

1.00 

0.75 

Conclusions 

The  partieipants  in  this  experiment  were  alike  in  their  experienee  levels,  serviee  affiliations,  and 
their  knowledge  of  FCS  teehnologies.  On  average,  the  group  had  9  years  of  enlisted  experienee 
in  the  Army. 

The  platoon  transeripts  demonstrate  that  network  eonneetivity  was  a  sporadie  but  pervasive 
problem.  The  dense  foliage  of  Ft.  Dix  and  the  hilly  terrain  undoubtedly  eontributed  to  network 
outages.  System  problems  also  contributed  to  platoon  efforts  to  prosecute  the  ISR  mission. 
Examples  of  these  inelude  loss  of  radio  eommunieations,  inability  of  the  UAV  to  launeh  on  time, 
and  inability  to  transmit  unmanned  sensor  images  on  a  eonsistent  basis.  Providing  the 
partieipants  more  time  to  praetiee  taetieal  missions  with  the  equipment  eould  have  alerted  us  to 
some  of  these  problems.  This  is  a  strong  reeommendation  for  future  trials. 

SA  was  generally  low  among  all  partieipants.  High  fidelity  seores  were  greater  in  the  advaneed 
ease  (15%)  eompared  to  the  base  case  (4%),  but  most  ratings  fell  in  the  low  and  medium  ranges. 
We  found  no  differenees  in  the  fidelity  scores  by  role,  suggesting  that  the  PL,  PSG  and  RNCO 
all  had  approximately  similar  levels  of  SA.  This  might  be  explained  by  the  tendeney  of  the  PL  to 
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station  these  players  in  close  proximity.  One  platoon  received  lower  fidelity  scores  than  the 
other.  We  are  at  a  loss  to  explain  this  trend  but  suspect  it  could  be  a  result  of  differences  in  the 
emphasis  on  data  collection.  The  accuracy  scores  for  the  PL  were  generally  uneven,  but  were 
slightly  higher  in  the  advanced  case. 

Most  test  participants  were  unable  to  answer  questions  about  the  enemy  force  and  were  unable  to 
locate  OPFOR  positions  on  a  map.  SA  did  improve  slightly  in  the  advanced  case,  when  leaders 
had  the  benefit  of  the  unmanned  sensor  images  and  the  FBCB2  tablets  for  dismounts. 
Unfortunately,  due  to  our  limited  data,  we  are  not  able  to  associate  improved  SA  with  the  use  of 
the  new  technologies.  The  noted  improvements  were  marginal  and  were  generally  restricted  to 
the  platoon  leader.  Two  possible  problems  may  account  for  the  low  levels  of  SA.  First,  data 
collected  on  the  SA  surveys  was,  at  times,  not  complete.  Providing  expanded  data  collector 
training  on  administering  the  surveys  and  improved  emphasis  with  the  Soldiers  on  the 
importance  of  this  type  of  survey  could  have  improved  the  completeness  of  the  data  collected. 

At  times,  the  Soldiers  completing  the  surveys  did  not  appear  to  put  much  effort  into  the  map 
annotations,  possibly  because  of  their  fatigue  or  their  lack  of  patience  for  completing  four 
surveys  each  day.  Some  data  collectors  who  were  responsible  for  the  SA  surveys  were  also 
responsible  for  other  activities,  possibly  causing  them  to  be  distracted  from  the  SA  survey 
administration. 

The  second  probable  cause  of  the  overall  low  SA  was  the  recurrent  problems  with  network 
connectivity  that  resulted  in  delayed  image  transfers  from  the  unmanned  assets  to  the  FBCB2 
displays.  These  problems  could  have  discouraged  the  platoon  leader  from  expending  his  time  on 
monitoring  the  images,  choosing  instead  to  focus  on  his  human  dismounts. 

The  FBCB2  tablets,  though  not  without  problems,  did  give  the  PL  a  better  view  of  how  the 
dismounted  troops  were  positioned  and  how  maneuvers  were  progressing.  Unfortunately,  the  PL 
became  so  involved  with  this  aspect  of  the  interface,  and  neglected  other  important  tasks  (such  as 
UAV  or  UGS  images  and  communications).  This  problem  could  have  been  moderated  through 
the  improved  use  of  the  RNCO  position.  TTPs  that  define  RNCO  duties  and  capabilities  are 
needed  and  should  be  developed  in  support  of  future  Testbed  experiments. 

Our  findings  suggest  that  while  unmanned  assets  clearly  provide  benefits  to  the  tactical  force,  the 
use  of  these  systems  extracts  a  cost  to  units  in  the  form  of  additional  physical  and  cognitive 
demands.  Continued  efforts  to  develop  robust  communication  networks  and  TTPs  for 
coordinating  these  systems  into  the  dismounted  force  should  be  aggressively  pursued.  A  primary 
focus  area  for  future  research  is  an  improved  understanding  of  how  unmanned  assets  can 
enhance  human  decision  making  in  uncertain  and  complex  terrain. 
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The  Unmanned  Sensor  Suite 


Platoon 
operated 
against  an 
unscripted 
OPFOR 


•  Buster  UAV 

•  Textron  Unmanned 
Ground  Sensors 

•  Small  Unmanned 
Ground  Vehicles 

•  Digitized  Battle 
Command  System: 
FBCB2 

•In  vehicles 

•  On  tablets  for 
dismounts 
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Objective:  Determine  the 
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Situation  Awareness  (SA) 
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data  storage  capability 
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Procedures 


The  Experiment  Mission  Day 


Convoy 
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Planning  and 
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AAR  and  equipment 
^  maintenance 
1600-1800 


Red  Force 
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Conditions 


/a 


Base  Case 

•  2  Platoons 

-  30  NJ  ARNG  Soldiers 

•  3  Vehicles/Platoon 

-  Platoon  Leader 

-  Platoon  Sergeant 

-  Robotics  NCO 

•  FBCB2  used  only  in  vehicles 

•  No  UAVs,  UGVs  used 

•  Unmanned  Sensors  used 

•  3  day  period 


Advanced  Case 

•  2  Platoons 

-  30  NJ  ARNG  Soldiers 

•  3  Vehicles  /  Platoon 

-  Platoon  Leader 

-  Platoon  Sergeant 

-  Robotics  NCO 

•  FBCB2  used  both  in  vehicles 
and  on  PC  tablets  for 
dismounts 

•  UAV,  UGV  used 

•  Unmanned  Ground  Sensors 
used 

•  5  day  period 


Robotics  NCO  used  in  both  cases 


005  -  DAY 
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Data  CollectioHr”:’ 

RNCO  J  P  r  2F7’C 


•  Demographics 

•  Situation  Awareness  Global  Assessment  Techniqtie 
(Endsley,  2000) 


-  11  questions 

;  J(  ■ 

2  -  Cover  spectrum  of  SA  levels 

•  Level  1  Perception  (what  is  happening) 

F  lens ‘he  Th  *  Level  2  Comprehension  (whatndoes  it  mean) 

K.  Does  the  i  •  Level  3  Projection  (what  will  happen  in  future) 

"  '  -  Map  notations 

•  Blue  vehicles  and  dismounts 

•  Red  vehicles  and  dismounts 

-  Administered  4  times  per  mission _ 

-  y  •  After  training  before  leaving  for  site  - 

■  '  '  •  Twice  during  the  mission  2  7  s  9  10 

2  _  ^  3  9  10 

^  •  Immediately  upon  mission  completion  ^  3  9  10 

2  7  3  9  ;  0 

-  Scored  based  upon  ground  truth  recordings  of  actual  Blue/Red 

locations  in  Testbed  data  base  ^  _ 


Demographic  Force  Comparisons 


Blue  Force:  NJ  ARNG  (30) 

OPFOR:  Goldbelt  Eagle  Contractor  (34  [minus  drivers]) 

Experience:  Blue  Force  had,  on  average,  9  years  of  military 
experience,  compared  to  OPFOR  average  of  3.4  years.  Of  the 
OPFOR  with  military  experience,  most  reported  Army  backgrounds 
(71%). 


Deployment  experience:  87%  of  Blue  Force  had  deployed  at  least 
once,  compared  to  24%  of  OPFOR.  Deployment  experience  was 
spread  in  BlueFor,  and  concentrated  among  a  few  in  OPFOR. 

Military  Experience  Comparisons 


Daily  Transcript  Analysis 


Category 

Messages 

Example 

Equipment  Breakdowns 

•  Radio  communications 

25 

“Today  was  a  challenge  for  communications  (vegetated  area).  5''"'  squad  could  hear  everyone, 
no  one  could  hear  them.  P‘  squad  comms  were  intermittent  at  best.  ’’ 

•  FBCB2  vehicle 

6 

“FBCB2  screen  keeps  cutting  out,  possibly  due  to  heat  from  engine.  ’’ 

•  FBCB2  tablet 

10 

“I  have  tablet  out,  but  can ’t  send  information.  When  I  put  the  pen  on  the  screen  it  is  4  to  5 
inches  to  the  left  of  where  I  need  to  mark,  the  cursor  moves  in  its  own  direction.  ” 

•  UAV 

9 

•  Engineers  trying  unsuccessfully  to  open  UA  V  spot  report  images,  believe  it  is  a  technical 
failure. 

•  Images  on  spot  report  are  blurred  to  point  of  being  unrecognizable 

•  SUGV 

1 

When  using  SUGV,  it  was  within  50  meters  of  target  vehicle,  but  didn ’t  detect  due  to  constraints 
of  sensor  movement. 

•  UGS 

1 

PL  is  advised  to  look  at  UGS  field.  Opens  one  image  (takes  13  seconds).  Opens  second  image 
(takes  17  seconds).  PL  misinterprets  retreating  OPFOR  vehicles  as  reinforcing  force. 

Time  pressure  and  lack  of  regular  monitoring  of  UGS  field  led  to  confusion  with  images. 

User  problems 

•  Dismounted  reports  to  leaders 

10 

“Any  2^“^  squad  element,  this  is  Platoon  Leader,  somebody  get  me  a  sit  rep.  ”  [from  2"^^  squad]: 
“Standby.  ”  “I  need  a  sit  rep  from  anyone.  Who ’s  next  in  that  chain  of  command?  Squad 

1,  is  there  anyone  left  alive  in  that  unit?  ’’ 

•  Use  of  subordinate  team 

leaders 

8 

Platoon  leader  stops  RNCO  from  downloading  UA  V  images  because  it  is  interfering  with  his 
attempts  to  download  and  view.  Lack  of  task  management. 

FBCB2  interface 

•  Spot  reports 

15 

All  squads  are  sending  spot  reports  to  PL  from  their  tablets. 

•  Free  text 

4 

PL  sends  priority  information  requirements  report  to  Company  Commander  in  free  text. 

•  Opening  images 

6 

Images  take  a  long  time  to  open,  often  will  not  open  at  all. 

8 


Roles  &  Responsibilities 

•  Platoons  operated  independently 

•  Leaders  had  difficulty  switching  between  tasks 
(FBCB2  and  sensor  images) 

•  FBCB2  Blue  Force  Tracking  became  a  cognitive 
tunnel  for  one  PL 

•  PLs  did  not  effectively  utilize  RNCO 

•  No  one  was  responsible  for  coordinating 
unmanned  sensors 

•  Examples  of  sensor  image  problems 

-  Are  vehicles  reinforcing  or  retreating? 

-  Where  is  the  UAV?  Does  the  UAV  know  our  location? 


9 


SA  Scoring:  Completeness 

•  Completeness  of  answers  (Fidelity) 

•  Scores  by  Case,  Role,  Platoon 


S. 


100% 

80% 

60% 

40% 

20% 

0% 


Case 


Low  Medium 

Levels  of  SA  Fidelity 


High 


■  Base 
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0) 

0. 


n  PL 
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Low 


Medium 
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High 


■  1st  Piatoon 
□  2nd  Piatoon 


Platoon 


Low 


Medium 

Levels  of  SA  Fidelity 


High 
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SA  Scoring:  Accuracy 

•  How  accurate  are 
the  locations  of 
Blue  and  Red  and 
activity 
descriptions? 

•  Most  SA  surveys 
had  0  or  very  low 
accuracy  ratings 

•  A  few  low  fidelity 
ratings  were 
accurate  (few  map 
annotations  but 
accurate) 

•  Five  high  fidelity 
surveys  had  0 
accuracy  ratings 

Accuracy  scores  based  on  SME  evaluation  of  errors  in  meters  of  map 
annotations  and  accuracy  of  answers 


SA  Fidelity  Rating 

SA 

Accuracy 

Scores 

Low  Fidelity 

N  (%) 

Medium  Fidelity 
N  (%) 

High  Fidelity 

N  (%) 

1.00 

1  (17%) 

3  (50%) 

2  (33%) 

0.75 

1  (20%) 

2  (40%) 

2  (40%) 

0.50 

1  (11%) 

8  (89%) 

0 

0.25 

0 

5  (100%) 

0 

0.00 

18  (69%) 

3  (12%) 

5  (19%) 

Conclusions 


•  No  noticeable  differences  between  Blue 
and  Red  Force  demographics 

•  Network  connectivity  a  sporadic  but 
pervasive  problem  that  impacted  radio 
communications  and  image  transmission 

•  SA  among  all  roles  was  generally  very  low 

•  SA  of  Blue  was  aided  in  advanced  case  by 
blue  force  tracker  capability  of  FBCB2 

•  Use  of  unmanned  systems  extracts  a 
cognitive  and  physical  cost  to  platoons 
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2006  Study  Plans 


Robotics  NCO  task  analysis 

-  Coordination  &  Control  of  unmanned  assets 

-  Coordination  of  unmanned  air  systems  to  provide 
NLOS  network  connections 

SAGAT  queries  in  30  minute  intervals  of  all 
Soldiers  by  embedded  data  collectors 

Scripted  OPFOR 

Increased  emphasis  on  pre-experiment  training 
in  Tactics,  Techniques,  and  Procedures 


